This paper argues for more randomised controlled trials in educational research. Educational researchers have largely abandoned the methodology they helped to pioneer. This gold-standard methodology should be more widely used as it is an appropriate and robust research technique. Without subjecting curriculum innovations to a RCT then potentially harmful educational initiatives could be visited upon the nation's children.

In response to concern over the numeracy skills deficit displayed by student nurses, an online computer programme, ?Authentic World??, which aims to simulate a real-life clinical environment and improve the medication dosage calculation skills of users, was developed (Founded in 2004 Authentic World Ltd is a spin out company of Glarmorgan and Cardiff Universities, Cardiff, Wales UK.). Two randomised controlled trials were conducted, each at a UK University, in order to investigate the impact of Authentic World? on student nurses? general (...) numeracy abilities. All first year nursing students who gave consent were randomised equally into an intervention or control group. The intervention group were given access to Authentic World?. The primary outcome measure was the students? scores on a general numeracy test. The Intention to Treat (ITT) analysis in both trials revealed a small negative effect of Authentic World? on general numeracy, which was statistically significant in one trial. However, compliance with the intervention was very low in both trials, with only 24 and 12% of students allocated to the intervention groups spending more than 15 minutes using the programme. Providing nursing students with access to Authentic World? is not an effective use of resources since use of the programme appears to be very low. (shrink)

Randomised controlled trials (RCTs) are often seen as the 'gold standard' of evaluative research. However, whilst randomisation will ensure comparable groups, trials are still vulnerable to a range of biases that can undermine their internal validity. In this paper we describe a number of common threats to the internal validity of RCTs and methods of countering them. We highlight a number of examples from randomised trials in education and health care where problems of execution and analysis of the RCT has (...) undermined their internal validity. However, awareness of these potential biases can lead to careful planning to avoid or reduce their occurrence. If good quality randomised trials are to inform policy and practice in education then rigorous trials need to be designed that are the least susceptible to threats to their validity. (shrink)

The aim of unpaid volunteer classroom assistants is to give extra support to children learning to read. The impact of using volunteers to improve children's acquisition of reading skills is unknown. To assess whether volunteers are effective in improving children's reading, we undertook a systematic review of all relevant randomised controlled trials (RCTs). An exhaustive search of all the main electronic databases was carried out (i.e. BEI, PsycInfo, ASSIA, PAIS, SSCI, ERIC, SPECTR, SIGLE). We identified eight experimental studies, of which (...) seven were RCTs. One of the RCTs was excluded because it did not meet the inclusion criteria. One RCT randomised intact classes and the other six studies randomised individual children and could therefore be included in a meta-analysis. All of the trials were fairly small, with the largest including 99 pupils. Four of the trials showed a positive outcome, while three showed a negative effect and the remaining study was equivocal. We pooled the four most homogeneous trials. The pooled data indicated an effect size of 0.19, which was not statistically significant ( p = 0.54, 95% confidence interval = -0.31 to 0.68). Overall, volunteering appeared to have a small effect on reading outcomes. However, the confidence intervals were wide, which could conceal a potentially large benefit or a harmful effect. Thus, more good quality RCTs are required in order to provide more conclusive evidence. (shrink)

ABSTRACT: The term ‘publication bias’ usually refers to the tendency for a greater proportion of statistically significant positive results of experiments to be published and, conversely, a greater proportion of statistically significant negative or null results not to be published. It is widely accepted in the fields of healthcare and psychological research to be a major threat to the validity of systematic reviews and meta-analyses. Some methodological work has previously been undertaken, by the author and others, in the field of (...) educational research to investigate the extent of the problem. This paper describes the problem of publication bias with reference to its history in a number of fields, with special reference to the area of educational research. Informal methods for detecting publication bias in systematic reviews and meta-analyses of controlled trials are outlined and retrospective and prospective methods for dealing with the problem are suggested. (shrink)

Randomized controlled trials in educational research tend to be small. Small trials can have large, chance, imbalances in important covariates. For studies with sample sizes greater than 50, chance imbalances can be corrected using analysis of covariance; for small trials, however, statistical power is maximized if the trial is balanced and analysis of covariance is used in the analysis. The aim of the present study was to discuss methods of improving covariate balance in trial design and to demonstrate the method (...) of minimization. Using an exemplar of a cluster or class‐randomized trial with 29 classes, we employed minimization to achieve covariate balance. Minimization achieved good balance on four prognostic variables. Many trialists in education use restricted forms of allocation, including pairing or stratified randomization. These approaches have disadvantages. Another approach rarely used in educational research is minimization. Minimization uses a simple arithmetic algorithm to produce balanced groups across a number of important covariates and should be more widely used in educational and psychological research. (shrink)